Newton-type methods for REML estimation in genetic analysis of quantitative traits

A. Robust and efficient optimization methods for variance component estimation using Restricted Maximum Likelihood (REML) models for genetic mapping of quantitative traits are considered. We show that the standard Newton-AI scheme may fail when the optimum is located at one of the constraint boundaries, and we introduce different approaches to remedy this by taking the constraints into account. We approximate the Hessian of the objective function using the average information matrix and also by using an inverse BFGS formula. The robustness and efficiency is evaluated for problems derived from two experimental data from the same animal populations.